Affective-cognitive load based digital assistant

ABSTRACT

Embodiments of the present disclosure sets forth a computer-implemented method comprising receiving, from at least one sensor, sensor data associated with an environment, computing, based on the sensor data, a cognitive load associated with a user within the environment, computing, based on the sensor data, an affective load associated with an emotional state of the user, determining, based on both the cognitive load at the affective load, an affective-cognitive load, determining, based on the affective-cognitive load, a user readiness state associated with the user, and causing one or more actions to occur based on the user readiness state.

BACKGROUND Field of the Various Embodiments

The various embodiments relate generally to digital assistants and, morespecifically, to an affective-cognitive load based digital assistant.

Description of the Related Art

Various digital systems include digital assistants that assist users toperform tasks. For example, various vehicles include an advanced driverassistance system (ADAS) that assists a driver in handling a vehicle.Such an ADAS may include a driver monitoring system (DMS) that monitorsthe driver in order to assess the driver's handling of the vehicle andenables the ADAS to respond to the state of the driver by providingvarious levels of assistance, such as by generating notifications,automating various driving tasks, etc.

A user typically splits focus between multiple objects when interactingwith a given environment. For example, a driver may split focus betweendriving a vehicle and conducting a conversation with another person. Incertain situations, the driver may not properly adjust to provide enoughfocus on driving tasks in order to adequately navigate complex drivingsituations that sporadically occur, such as an increase in traffic,adverse weather conditions, sudden obstacles, and so forth.

In general, an ADAS may easily detect distracting objects and maydetermine whether a driver is focused on the road. However, the drivermonitoring system may not be able to determine the amount of focus thedriver has when performing driving tasks. For example, the DMS mayinclude a camera that is used to determine whether the driver's eyes arefocused on the road, but the DMS may not account for other objects andstimuli that otherwise lessen the focus on successfully performingdriving tasks. As a result, ADAS may not be able to effectively assistthe driver and the driver may fail to modify their behavior to properlyhandle the necessary driving tasks.

In light of the above, more effective techniques for monitoring thestatus of the user interacting in an environment would be useful.

SUMMARY

One embodiment sets forth a computer-implemented method comprisingreceiving, from at least one sensor, sensor data associated with anenvironment, computing, based on the sensor data, a cognitive loadassociated with a user within the environment, computing, based on thesensor data, an affective load associated with an emotional state of theuser, determining, based on both the cognitive load at the affectiveload, an affective-cognitive load, determining, based on theaffective-cognitive load, a user readiness state associated with theuser, and causing one or more actions to occur based on the userreadiness state.

Further embodiments provide, among other things, a method and a systemconfigured to implement the computer-readable storage medium set forthabove.

At least one technological advantage of the disclosed techniques overprevious digital assistant systems is that computing a user readinessstate based on a combination of direct measurements that estimate thecognitive load of a user and the emotional state of the user provides amore accurate indication of the ability of the user to handle one ormore tasks.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the variousembodiments can be understood in detail, a more particular descriptionof the inventive concepts, briefly summarized above, may be had byreference to various embodiments, some of which are illustrated in theappended drawings. It is to be noted, however, that the appendeddrawings illustrate only typical embodiments of the inventive conceptsand are therefore not to be considered limiting of scope in any way, andthat there are other equally effective embodiments.

FIG. 1 illustrates a block diagram of the user readiness systemconfigured to implement one or more aspects of the present disclosure.

FIG. 2 illustrates a passenger compartment of a vehicle that isconfigured to implement the user readiness system of FIG. 1, accordingto various embodiments.

FIG. 3 illustrates a block diagram of the user readiness application ofFIG. 1, according to various embodiments.

FIG. 4A illustrates an example lookup table of affective-cognitive loadvalues associated with the user readiness application of FIG. 1,according to various embodiments.

FIG. 4B illustrates another example lookup table of affective-cognitiveload values associated with the user readiness application of FIG. 1,according to various embodiments.

FIG. 5 illustrates an affective-cognitive load computed from variousbiometric values derived by the user readiness application of FIG. 1,according to various embodiments.

FIG. 6 illustrates an example vehicle system that includes anaffective-cognitive load based digital assistant of FIG. 3, according tovarious embodiments.

FIG. 7 is a flow diagram of method steps for generating an output signalbased on an affective-cognitive load, according to various embodiments.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth toprovide a more thorough understanding of the various embodiments.However, it will be apparent to one of skilled in the art that theinventive concepts may be practiced without one or more of thesespecific details.

System Overview

FIG. 1 illustrates a block diagram of user readiness system 100configured to implement one or more aspects of the present disclosure.As shown, user readiness system 100 includes, without limitation,computing device 110, sensor(s) 120, input/output (I/O) device(s) 130,and network 150. Computing device 110 includes processing unit 112 andmemory 114, where memory 114 stores user readiness application 140 anddatabase 142. Network 150 includes data store 152.

In operation, processing unit 112 receives sensor data from sensor(s)120. Processing unit 112 executes user readiness system 140 to in orderto process the sensor data and determine various biometric valuesassociated with the psychophysiological state a user. Such biometricvalues include a cognitive load that estimates an amount of brainactivity that a user is employing, and an affective load that estimatesan emotion (specified as a pre-defined emotion or a set of emotionparameterized metrics associated with a psychophysiological state) thata user is experiencing. In various embodiments, the affective load mayinclude one or more separate emotional metrics, such as separate arousaland/or valence values. Additionally or alternatively, the affective loadmay comprise one or more discrete emotional states and/or associatedvalues of each. Upon determining biometric values for a user, userreadiness application 140 combines the biometric values via one or moreestimation algorithms to determine an affective-cognitive load (ACL) asa composite of the biometric values. In various embodiments, userreadiness application 140 compares the ACL with other ACL data includedin database 142 and/or data store 152 in order to map the ACL to a userreadiness that estimates the ability of the user to handle one or moretasks. User readiness application 140 and/or other applications (notshown) may then generate output signals based on the determined userreadiness state. For example, user readiness application 140 may be acomponent of an advanced driver assistance system (ADAS) that automatescertain tasks and/or provides notifications via one or more outputsignals once user readiness application 140 determines that the userreadiness is outside of a target value and/or target range.

As noted above, computing device 110 can include processing unit 112 andmemory 114. Computing device 110 can be a device that includes one ormore processing units 112, such as a system-on-a-chip (SoC). In variousembodiments, computing device 110 may be a mobile computing device, suchas a tablet computer, mobile phone, media player, and so forth. In someembodiments, computing device 110 may be a head unit included in avehicle system. Generally, computing device 110 can be configured tocoordinate the overall operation of user readiness system 100. Theembodiments disclosed herein contemplate any technically-feasible systemconfigured to implement the functionality of user readiness system 100via computing device 110.

Various examples of computing device 110 include mobile devices (e.g.,cellphones, tablets, laptops, etc.), wearable devices (e.g., watches,rings, bracelets, headphones, etc.), consumer products (e.g., gaming,gambling, etc.), smart home devices (e.g., smart lighting systems,security systems, digital assistants, etc.), communications systems(e.g., conference call systems, video conferencing systems, etc.), andso forth. Computing device 110 may be located in various environmentsincluding, without limitation, home environment (e.g., office, kitchen,etc.), road vehicle environments (e.g., consumer car, commercial truck,etc.), aerospace and/or aeronautical environments (e.g., airplanes,helicopters, spaceships, etc.), nautical and submarine environments, andso forth.

Processing unit 112 may include a central processing unit (CPU), adigital signal processing unit (DSP), a microprocessor, anapplication-specific integrated circuit (ASIC), a neural processing unit(NPU), a graphics processing unit (GPU), a field-programmable gate array(FPGA), and so forth. Processing unit 112 generally comprises aprogrammable processor that executes program instructions to manipulateinput data. In some embodiments, processing unit 112 may include anynumber of processing cores, memories, and other modules for facilitatingprogram execution. For example, processor unit 112 could receive inputfrom a user via I/O devices 130 and generate pixels for display on I/Odevice 130 (e.g., a display device). In some embodiments, processingunit 112 can be configured to execute user readiness application 140 inorder to analyze acquired sensor data and estimate the ability of a userto handle specific tasks by determining a user readiness state. In suchinstances, user readiness application 140 may output the estimated userreadiness state to one or more I/O modules 130 in order for the I/Omodules 130 that respond to the determined user readiness state (e.g.,output signals to provide assistance, output signals to enable a user toperform manual tasks, etc.).

Memory 114 can include a memory module or collection of memory modules.Memory 114 generally comprises storage chips such as random accessmemory (RAM) chips that store application programs and data forprocessing by processing unit 112. In various embodiments, memory 114may include non-volatile memory, such as optical drives, magneticdrives, flash drives, or other storage. In some embodiments, separatedata stores, such as data store 152 included in network 150 (“cloudstorage”) may supplement memory 114. User readiness application 140within memory 114 can be executed by processing unit 112 to implementthe overall functionality of the computing device 110 and, thus, tocoordinate the operation of the user readiness system 100 as a whole.

User readiness application 140 processes acquire sensor data associatedwith a user and/or environment in order to determine various metricsassociated with the user's brain activity and/or emotion. In variousembodiments, user readiness application 140 may receive sensor data fromone or more sensors 120 and may analyze the sensor data in order todetermine the cognitive load of the user and/or the affective load,which may include separate emotional parameters that indicate theemotion being experienced by the user. User readiness application 140determines, based on both the cognitive load and the affective load, anaffective-cognitive load (ACL) of the user that indicates the user'soverall mental activity and ability to manage a set of tasks. In variousembodiments, user readiness application 140 may compare the ACL withother ACL values and/or thresholds in order to map the ACL to a specificuser readiness state. User readiness application 140 may then output theuser readiness state to one or more I/O devices 130 that respond to thedetermined user readiness state. For example, user readiness application140 may determine that a current user readiness state is below thethreshold level required to successfully navigate the current trafficconditions. User readiness application 140 could then send the userreadiness state an ADAS that sends an output signal to I/O device 130(e.g., a display device) in order to play a notification sound to alertthe driver. In some instances, the ADAS could also respond by performingone or more automated tasks (e.g., assisted driving).

Database 142 can store values and other data retrieved by processingunit 112 to coordinate the operation of user readiness application 140.In various embodiments, processing unit 112 may be configured to storevalues in database 142 and/or retrieve values stored in database 142.For example, database 142 could store historical ACL values, lookuptables, ACL algorithms, mappings of sensor values to emotionalparameterized metrics, mappings of ACL values to driver readinesslevels, and so forth. In some embodiments, database 142 may store valuesretrieved from data store 152. In such instances, database 142 mayreceive periodic updates and provide values to user readinessapplication 140 between the periodic updates.

In various embodiments, database 142 can include one or more lookuptables, where the lookup tables store entries that include mappingsbetween values. For example, database 142 could include a set of ACLlookup tables that includes entries of mappings of biometric values(e.g., cognitive load, affective load, arousal, valence, etc.), to ACLvalues. Additionally or alternatively, database 142 can include a set ofACL lookup tables that maps biometric values to pre-defined ACL levels(e.g., high ACL, medium ACL, low ACL, etc.), and/or a set of ACL lookuptables that maps pre-defined biometric values (e.g., a defined value forthe psychophysiological trait of “angry”) to specific ACL values and/orpre-defined ACL levels.

Sensor(s) 120 may include one or more devices that perform measurementsand/or acquire data related to certain subjects in an environment. Invarious embodiments, sensor(s) 120 may generate sensor data that isrelated to the cognitive load and/or affective load of the user. Forexample, sensor(s) 120 could collect biometric data related to the user(e.g., heart rate, brain activity, skin conductance, blood oxygenation,pupil size, galvanic skin response, blood-pressure level, average bloodglucose concentration, etc.). Additionally or alternatively, sensor(s)120 can generate sensor data related to objects in the environment thatare not the user. For example, sensor(s) 120 could generate sensor dataabout the operation of a vehicle, including the speed of the vehicle,pedal position, steering wheel position, ambient temperature in thevehicle, amount of light within the vehicle, and so forth. In someembodiments, sensor(s) 120 may be coupled to and/or included withincomputing device 110 and send sensor data to processing unit 112.Processing unit 112 executes user readiness application 140 in order todetermine a user readiness state based on a determinedcognitive-affective load that is derived from the acquired sensor data.

In various embodiments, sensor(s) 120 may acquire sensor data that userreadiness application 140 processes in order to classify an emotion thatthe user is experiencing. For example, sensor(s) 120 could include auser-facing camera that records the face of the user as image data. Userreadiness application 140 could then analyze the image data in order todetermine the facial expression of the user, and then map the facialexpression to a specific emotion. In another example, sensor(s) 120could include sensors in various parts of the vehicle (e.g., driver'sseat passenger seat, steering wheel, etc.) that acquire biologicaland/or physiological signals of a user (e.g., perspiration, heart rate,heart-rate variability (HRV), blood flow, blood-oxygen levels, breathingrate, galvanic skin response (GSR), sounds created by a user, behaviorsof a user, etc.). In such instances, user readiness application 140could compute one or more quantitative emotional parameterized metrics,such as emotional arousal (A) and/or emotional valence (V) that indicatethe emotion the user is experiencing.

In various embodiments, the sensor(s) 120 may also acquire data thatuser readiness application 140 processes in order to compute a cognitiveload that a user is experiencing. For example, sensor(s) 120 couldinclude a pupil sensor (e.g., a camera focused on the eyes of the user)that acquires image data about at least one pupil of the user. Userreadiness application 140 could then perform various pupillometrytechniques to detect eye parameters (e.g., fluctuations in the user'spupil diameter, direction of the pupil is gazing, eye lid position,etc.) in order to estimate a cognitive load (CL) of the user. In anotherexample, sensor(s) 120 could include heart rate sensors and/or otherbiometric sensors that acquire biological and/or physiological signalsof the user (e.g., heart rate, breathing rate, eye motions, GSR, neuralbrain activity, etc.). In such instances, user readiness application 140could compute the cognitive load from one or more of the acquiredbiological and/or physiological signals.

In various embodiments, the sensor(s) 120 may include optical sensors,such as RGB cameras, infrared cameras, depth cameras, and/or cameraarrays, which include two or more of such cameras. Other optical sensorsmay include imagers and laser sensors. In some embodiments, sensor(s)120 may include physical sensors, such as touch sensors, pressuresensors, position sensors (e.g., an accelerometer and/or an inertialmeasurement unit (IMU)), motion sensors, and so forth, that register thebody position and/or movement of the user. In such instances, userreadiness application 140 may analyze the acquired sensor data todetermine the movement of the user, and then correlate such movementwith specific emotions (e.g., boredom, fatigue, arousal, etc.) and/or acognitive load of the user.

In various embodiments, the sensor(s) 120 may include physiologysensors, such as heart-rate monitors, electroencephalography (EEG)systems, radio sensors, thermal sensors, galvanic skin response sensors(e.g., sensors that measure change in electrical resistance of skincaused by emotional stress), contactless sensor systems,magnetoencephalography (MEG) systems, and so forth. In variousembodiments, user readiness application 140 may execute spectralentropy, weighted mean frequency, bandwidth, and/or spectral edgefrequency to determine cognitive load from the acquired sensor data.

In addition, in some embodiments, sensor(s) 120 may include acousticsensors, such as a microphone and/or a microphone array that acquiressound data. Such sound data may be processed by user readinessapplication 140 performing various natural language (NL) processingtechniques, sentiment analysis, and/or speech analysis in order todetermine the semantic meaning of the phrases spoken in the environmentand/or infer emotional parameterized metrics from the semantic meaning.In another example, user readiness application 140 could analyze theacquired sound data using voice-tone analysis in order to infer emotionfrom the speech signal included in the sound data. In some embodiments,user readiness application 140 may execute various analysis techniquesrelating to the spectral centroid frequency and/or amplitude of thesound signal in order to classify the sound signal to a specific valuefor the cognitive load.

In some embodiments, sensor(s) 120 may include behavioral sensors thatdetect the activity of the user within the environment. Such behavioralsensors may include devices that acquire related activity data, such asdevices that acquire application usage data and/or mobile device usagedata. In such instances, user readiness application 140 may estimate thecognitive load and/or the emotional parameterized metrics by determiningthe activities in which the user is currently engaged. For example, agiven application could be classified as being a fun, social applicationin which a user engages when happy and active, and/or is making the userhappy and active. In such instances, user readiness application 140could correlate the usage of the given application with a pre-definedemotion (e.g., excited) and/or pre-defined emotional parameterizedmetrics (a high arousal value and a positive valence value).

I/O device(s) 130 may include devices capable of receiving input, suchas a keyboard, a mouse, a touch-sensitive screen, a microphone and otherinput devices for providing input data to computing device 110. Invarious embodiments, I/O device(s) 130 may include devices capable ofproviding output, such as a display screen, loudspeakers, and the like.One or more of I/O devices 130 can be incorporated in computing device110 or may be external to computing device 110. In some embodiments,computing device 110 and/or one or more I/O device(s) 130 may becomponents of an advanced driver assistance system. In variousembodiments, user readiness application 140 may determine a cognitiveload and/or an emotional load based on inputs received by one or moreI/O devices 130. For example, the vehicle could include a head unit thatincludes a user interface. In such instances, user readiness application140 could determine the cognitive load and/or the emotional load of theuser based on one or more inputs received via the head unit.

Network 150 may enable communications between computing device 110 andother devices in network via wired and/or wireless communicationsprotocols, satellite networks, V2X networks, including Bluetooth,Bluetooth low energy (BLE), wireless local area network (WiFi), cellularprotocols, and/or near-field communications (NFC). In variousembodiments, network 150 may include one or more data stores 152 thatstore data associated with sensor data, biometric values,affective-cognitive loads, and/or driver readiness levels. In variousembodiments, user readiness application 140 and/or other digitalassistants included in other devices may retrieve information from thedata store 152. In such instances, user readiness application 140 mayanalyze the retrieved data as part of determining the ACL of the user,comparing the ACL to situations that involve specific ACL values, and soforth.

FIG. 2 illustrates a passenger compartment 200 of a vehicle that isconfigured to implement the user readiness system 100 of FIG. 1,according to various embodiments. As shown, passenger compartment 200includes, without limitation, dashboard 210, windshield 220, and headunit 230. In various embodiments, passenger compartment 200 may includeany number of additional components that implement anytechnically-feasible functionality. For example, passenger compartment200 could include a rear-view camera (not shown).

As shown, head unit 230 is located in the center of dashboard 210. Invarious embodiments, head unit 230 may be mounted at any location withinpassenger compartment 200 in any technically-feasible fashion that doesnot block the windshield 220. Head unit 230 may include any number andtype of instrumentation and applications and may provide any number ofinput and output mechanisms. For example, head unit 230 could enableusers (e.g., the driver and/or passengers) to control entertainmentfunctionality. In some embodiments, head unit 230 may include navigationfunctionality and/or advanced driver assistance (ADA) functionalitydesigned to increase driver safety, automate driving tasks, and soforth.

Head unit 230 supports any number of input and output data types andformats, as known in the art. For example, head unit 230 could includebuilt-in Bluetooth for hands-free calling and/or audio streaming,universal serial bus (USB) connections, speech recognition, rear-viewcamera inputs, video outputs for any number and type of displays, andany number of audio outputs. In general, any number of sensors (e.g.,sensor(s) 120), displays, receivers, transmitters, etc., may beintegrated into head unit 230, or may be implemented externally to headunit 230. In various embodiments, external devices may communicate withhead unit 230 in any technically-feasible fashion.

While driving, the driver of the vehicle is exposed to a variety ofstimuli that are related to either a primary task (e.g., guiding thevehicle) and/or any number of secondary tasks. For example, the drivercould see via windshield 220, lane markers 240, cyclist 242, aspecialized car 244, and/or pedestrian 246. In response, the drivercould steer the vehicle to track lane markers 240 while avoiding cyclist242 and pedestrian 246, and then apply the brake pedal to allow policecar 244 to cross the road in front of the vehicle. Further, the drivercould concurrently or intermittently participate in conversation 250,listen to music 260, and/or attempt to soothe crying baby 270.

As shown, differing driving environments and/or engagement in secondaryactivities (e.g., deep thinking, emotional conversations, etc.)typically increase the cognitive load experienced by the driver and maycontribute to an unsafe driving environment for the driver and for inthe proximity of the vehicle. In addition, the emotion experienced bythe driver may also contribute to the unsafe driving environment, as theemotion that the user is experiencing may increase or decrease theability of the driver to handle driving tasks.

The Affective-Cognitive Load Based Digital Assistant

FIG. 3 illustrate affective-cognitive load-based assistant 300 includedin the user readiness system 100 of FIG. 1, according to variousembodiments. As shown, affective-cognitive load-based assistant 300includes sensor(s) 120, user readiness application 140, and outputdevice 340. User readiness application 140 includes biometriccomputation module 320 (including cognitive load computation module 314and emotion computation module 316) and user readiness computationmodule 330.

In operation, biometric computation module 320 includes various modules314, 316 that analyze sensor data 322 received from sensor(s) 120 inorder to determine one or more biometric values, including cognitiveload 324 and/or emotion metrics 326. Biometric computation module 320sends biometric values 324, 326 to user readiness computation module 330that performs one or more algorithmic techniques in order to determinean affective-cognitive load (ACL) of the user. User readinesscomputation module 330 may then map the ACL onto user readiness state332 that indicates the ability of the user to handle tasks. Userreadiness computation module 330 sends the user readiness state 332 toone or more output device(s) 340 that provide output signal(s) 342 thataffect the user and/or modify the behavior of the vehicle. For example,user readiness application 140 could send user readiness state 332 toseparate output devices 340, including to an ADAS and/or to a separateapplication on a device associated with the driver (e.g., an applicationrunning on a wearable device). In such instances, both output devices340 could respond by providing output signals 342, such as anotification signal that causes the wearable device to provide a hapticresponse. In some instances, an output device may directly affect thevehicle, such as when output signals 342 include one or more drivingadjustment signals that adjust the direction and/or speed of thevehicle.

In various embodiments, biometric computation module 320 receives sensordata 322 from sensor(s) 120. Biometric computation module may includeseparate modules that analyze portions of sensor data 322 in order toprovide cognitive load 324 and/or emotion metrics 326 (e.g., arousal andvalence). In some embodiments, modules 314, 316 may analyze the sameportions of sensor data. For example, cognitive load computation module314 and emotion computation module 316 may separately receive image dataincluded in sensor data 322. In such instances, cognitive loadcomputation module 314 may analyze the image data using variouspupillometry techniques and/or eye motion data to determine cognitiveload 324, while emotion computation module 316 may analyze the imagedata to determine the facial expression, classify the facial expressionas a pre-defined emotion, and acquire emotion metrics 326 correspondingto the pre-defined emotion.

User readiness computation module 330 applies various algorithms todetermine an affective-cognitive load as a function of cognitive load324 (CL) and the emotion metrics 326 that form an affective load (AL),where ACL=f (CL, AL). In various embodiments, a set of ACLs may bestored in database 142. In some embodiments, user readiness computationmodule 330 may refer to one or more lookup tables in order to find anentry corresponding to the values for cognitive load 324 and emotionmetrics 326 and may determine a corresponding ACL included in the entry.Additionally or alternatively, upon determining the ACL, user readinesscomputation module 330 may determine a user readiness state 332applicable to the determined ACL. In some embodiments, the lookup tablemay also include an applicable action that is to be taken for thecorresponding ACL. For example, a lookup table entry corresponding to anegative valence value, high arousal, and high cognitive load mayindicate a high ACL and may also include commands for the ADAS tooverride the manual driving actions performed by the user.

In various embodiments, user readiness computation module 330 may usevarious algorithmic techniques to compute an ACL value from one or moreof cognitive load 324 (CL) and/or emotion metrics 326, including avalence value (V), and an arousal value (A). In such instances, userreadiness computation module 330 may compare the ACL value to one ormore thresholds in order to determine user readiness state 332. In someembodiments, user readiness computation module 330 may select a specificalgorithm to employ in order to emphasize one metric relative to othermetrics. For example, user readiness computation module 330 couldemphasize cognitive load 324 relative to emotion metrics 326 by applyingequation 1 or equation 2 (where the offset value modifies the ACLrelative to a specific threshold). In such instances, the squared valueof CL causes cognitive load 324 to be the major indicator of ACL.

$\begin{matrix}{{ACL} = {\left( {\frac{1}{CL^{2}} \times \frac{1}{A}} \right) \times V}} & (1) \\{{ACL} = {\left( {\frac{1}{\left( {{CL} - {0.5}} \right)^{2}} \times \frac{1}{A}} \right) \times V}} & (2)\end{matrix}$

In another example, user readiness computation module 330 could applyequation 3 such that the ACL is based on a combination of cognitive load324 and the arousal value, which is modified by whether the valencevalue is positive or negative.

ACL=(CL+A)×V   (3)

In some embodiments, user readiness computation module 330 weighscertain values in order to emphasize a specific range of values. Forexample, user readiness computation module 330 could apply equation 4 or5 in order to emphasize that positive valence values are more desirablethan negative valence values.

$\begin{matrix}{{ACL} = {\left( {\frac{1}{CL} + \frac{1}{A}} \right) \times V}} & (4) \\{\frac{1}{ACL} = {{CL} + {A \times \left( \frac{1}{V + 1} \right)}}} & (5)\end{matrix}$

In some embodiments, one metric may be excluded. For example, userreadiness computation module 330 could apply equation 6 such that theACL is based on cognitive load 324 and the arousal value while ignoringthe valence value.

$\begin{matrix}{\frac{1}{ACL} = {{CL} + A}} & (6)\end{matrix}$

In various embodiments, user readiness computation module 330 may adjustthe selected algorithm and/or equation when computing the ACL. In suchinstances, user readiness computation module 330 may periodically selecta different algorithm and/or different equation. For example, whentraffic is light, user readiness module 330 can initially select analgorithm that emphasizes cognitive load 324. When affective-cognitiveload-based assistant 300 subsequently determines that traffic is heavy,user readiness computation module 330 may then select a differentalgorithm that weighs valence values more heavily and/or indicates thatpositive valence values are more desirable than negative valence values.

In some embodiments, user readiness computation module 330 mayincorporate other measurements associated with the user. In someembodiments, user readiness computation module 330 may derive measuresfocus and engagement by simultaneously analyzing a combination ofcognitive and emotional information. For example, user readinesscomputation module 330 could employ various statistical methods,machine-learning (ML) methods, state machines, and/or various other datastructures in order to determine how results of focus and engagement(and/or other user metrics), are derived from cognitive load 324, and/oremotion metrics 326.

Additionally or alternatively, ML models may be trained from anycombination of cognitive load 324, emotion metrics 326, and/or othermetrics. The ML models may be trained to predict specific higher-orderstates of the user, such as a driver attending to a specific drivingtask. The ML model may then be able to generate values, such as anengagement metric, to output module 340. For example, an ADAS couldreceive an engagement metric, modeled from affective and cognitivesignals, indicating the engagement level of a driver. In such instances,the engagement metric may be used by user readiness application 140and/or other applications or devices to affect the control of vehicle,such that when the drivers are less engaged, certain components like(e.g., ADAS functionalities) are more proactive.

Output module 340 receives user readiness state 332, which representsthe ability of the user to make a correct decision at the right timewhen performing a task. In certain situations, such as driving, userreadiness state 332 indicates how ready the driver is to drive or act inthe correct manner in a driving situation. In various embodiments,output device 340 generates output signals 342 based on user readinessstate 332. For example, when user readiness state 332 indicates that theuser is capable of handling a task within the environment, outputsignals 342 may cause one or more devices to execute tasks or provideoutputs that maintain the current state of the user (e.g., maintainplayback of an audio track). Conversely, when user readiness state 332indicates that the user is not capable of handling a task within theenvironment, output signals 342 may cause one or more devices to executetasks or provide outputs that assist the user in performing therequisite tasks.

In some embodiments, user readiness state 332 may indicate an expectedstate of the user at a future point. For example, in some embodiments,user readiness application 140 could execute various predictivetechniques to determine that, based on current input values, a predicteduser readiness state. In such instances, user readiness application 140could determine whether the user will need assistance at the futurepoint. In such instances, user readiness application 140 could generateone or more output signals 342 that modify the one or more components ofthe vehicle in anticipation of the user's state by the future point,thereby assisting the user in anticipation of the user's expectedreadiness state.

Determining User Readiness from Affective-Cognitive Load

FIG. 4A illustrates example lookup tables of affective-cognitive loadvalues associated with the user readiness application 140 of FIG. 1,according to various embodiments. As shown, tables 400 include lookuptable 410 for positive valence values and lookup table 430 for negativevalence values. FIG. 4B illustrates another example lookup table ofaffective-cognitive load values associated with the user readinessapplication of FIG. 1, according to various embodiments. As shown,tables 440 include lookup table 450 for positive valence values andlookup table 460 for negative valence values. In each lookup table 400,450, a given entry is coded as either a ready state (e.g., light grey),some emotional and cognitive load (e.g., medium grey) , significantemotional and cognitive load (e.g., dark grey), and/or high emotionaland cognitive load (e.g., black).

Lookup tables 400 represent conditions where the target range of the ACLis low. For example, lookup tables 400 could correspond to conditionswhere user performance is more effective as the ACL lowers. For example,when the ACL stems mainly from secondary tasks (e.g., texting, singing,etc.), lookup tables 440 represent conditions where the target range ofthe ACL is medium. For example, lookup tables 440 correspond toconditions where user performance is more effective as within a certainrange. For example, when a user is engaging in too little cognitivestimulation and therefore a low ACL (e.g., bored, tired, etc.), the ACLindicates that driver readiness 332 decreases not only for high ACL, butalso for low composite CL. In various embodiments, a range of high ACLvalues is the least desirable range and requires the most interventionby affective-cognitive load assistant 300.

In operation, user readiness computation module 330 may use cognitiveload 324 and emotion metrics 326 (e.g., a specific valence value and aspecific arousal value) in order to determine a specific entry in table410 or 430. In such instances, each metric is separated into two or morediscrete states. Lookup tables 410, 430 reflect certain heuristicsbetween specific combinations of values, ACL, and user readiness state332. For example, a combination of high cognitive load 324, a positiveemotion (positive valence), and a high amount of emotion (high arousal)is less detrimental to user performance than a combination of highcognitive load 324, a negative emotion (negative valence), and a highamount of emotion. In such instances, the entry for the firstcombination may indicate a lower ACL and result in a user readinessstate 332 within a target range (shown as entry 412), while the secondcombination may indicate a higher ACL and result in a user readinessstate 332 outside the target range (shown as entry 432).

In some embodiments, lookup tables 410, 430 may be generated fromhistorical data associated with a plurality of users. In such instances,database 142 the most recent lookup tables 410, 430 and may be referredto by user readiness computation module 330. In some embodiments, userreadiness application 140 may adjust one or more of lookup tables 410,430 based on one or more adjustment weights associated with a user. Forexample, user readiness application 140 could perform various baseliningand/or ML techniques using historical ACL values and performance metricsas training data in order to apply weights to ACL determinationalgorithms. For example, user readiness application 140 could analyzepersonalized user data and determine that a particular user requires aminimum ACL and/or a negative valence value in order to perform certaindriving tasks effectively. In such instances, user readiness application140 may adjust one or more thresholds and/or one or more entriesincluded in lookup tables 410, 430 and store the personalizedadjustments in database 142.

FIG. 5 illustrates an affective-cognitive load computed from variousbiometric values derived by the user readiness application 140 of FIG.1, according to various embodiments. As shown, graph 500 illustrates ACLis a function of cognitive load 324 levels for a given arousal value.

In various embodiments, user readiness computation module 330 may applyequation 7 or 8 in order to compute ACL as a continuous function along aWeibull distribution.

ACL=A×CL^(A) ×e ^(−CL) ^(A)   (7)

ACL=CL×A^(CL) ×e ^(−A) ^(CL)   (8)

Line 510 illustrates a Weibull distribution of ACLs corresponding to arange of cognitive loads and a low arousal value. For low arousallevels, the ACL remains approximately constant for a range of cognitiveloads 324 (except for very-low cognitive loads). In such instances, theACL never rises above a certain upper limit, indicating that the driverreadiness state for low arousal values never reaches the maximum value.For example, reaction times for a user could be relatively slower forlow arousal values.

Line 520 illustrates a Weibull distribution of ACLs corresponding to arange of cognitive loads and a medium arousal value. For medium arousallevels, the ACL and associated user readiness state 332 may be higherfor a certain range of composite loads 324. For example, above a certaincognitive load 324, the ACL for medium arousal values is higher than ACLvalues for lower arousal levels, indicating that the active mind of theuser enables the user to handle more-demanding tasks. Once cognitiveload 324 reaches an upper threshold, the cognitive ACL and userreadiness state 332 begin to decrease (e.g., secondary tasks distractingthe user).

Line 530 illustrates a Weibull distribution of ACLs corresponding to arange of cognitive loads and a high arousal value. When in a highemotional state, the range of best-possible performance (e.g., highestACL values) is narrower than with lower arousal levels, as seen by lines510, 520. Further the maximum ACL and associated user readiness state332 are significantly lower than the maximum ACL associated with lowerarousal levels.

FIG. 6 illustrates an example vehicle system 600 that includes anaffective-cognitive load digital assistant 300 of FIG. 3, according tovarious embodiments. As shown, vehicle system 600 includes sensingmodule 620, head unit 230, network 150, and output module 340. Sensingmodule 620 includes driver-facing sensors 622 (e.g., a camera),compartment non-driver-facing sensors 624 (e.g., steering wheel sensors,pedal sensors, etc.), and vehicle sensors 626 (e.g., speedometer,accelerometer, etc.). Head unit 230 includes entertainment subsystem612, navigation subsystem 614, network module 616, and advanced driverassistance system (ADAS) 618. Output module 340 includes ADASnotifications 642, ADAS parameters 644, human-machine interface (HMI)646, vehicle behaviors 648, application parameters 652, and applicationevents 654.

In various embodiments, user readiness application 140 may be includedin ADAS 618 and may receive sensor data from one or more sensors 622,624, 626 included in sensing module 620. In various embodiments, userreadiness application 140 may further receive data from navigationsubsystem 614 and/or network module 616. User readiness application 140analyzes the received data to compute an ACL and determine a userreadiness state 332 associated with the cognitive load and emotionalstate of the driver.

In various embodiments, ADAS 618 may send the user readiness state 332to output module 340 that generates one or more output signals. Inalternative embodiments, ADAS 618 may generate output signals based onthe user readiness state 332. In various embodiments, the output signalmay include one or more of ADAS notification 642, ADAS parameter 644,vehicle behavior 648, application parameter 652, and/or applicationevent 654.

Sensing module 620 includes multiple types of sensors, includingdriver-facing sensors 622 (e.g., cameras, motion sensors, etc.),compartment non-driver facing sensors 624 (e.g., motion sensors,pressure sensors, temperature sensors, etc.), and vehicle sensors (e.g.,outward-facing cameras, accelerometers, etc.). In various embodiments,sensing module 620 provides a combination of sensor data that describesthe context in which combined affective-cognitive load are beingobserved in more detail. For example, sensing module 620 could provide aset of values associated with the operation of the vehicle (e.g.,angular velocity of rear tires, velocity of the pedal movement, velocityof the vehicle, etc.). In such instances, user readiness application 140could determine a cognitive load value and/or an emotional load valuebased on the received values, such as by comparing the measured velocityof the vehicle compared to the speed limit of the location, and/or thevelocity of surrounding vehicles.

In various embodiments, vehicle sensors 626 may further include otherexternal sensors. Such external sensors may include optical sensors,acoustic sensors, road vibration sensors, temperature sensors, etc. Insome embodiments, sensing module and/or network module 616 may acquireother external data, such as geolocation data (e.g., GNNS systems,including a global positioning system (GPS), Glonass, Galileo, etc.). Insome embodiments, navigation data and/or geolocation data may becombined to predict changes to the ACL based on expected drivingconditions. For example, an expected traffic jam may cause userreadiness application 140 to predict an increase in the ACL upon thevehicle reaching affected area.

Network module 616 translates results of sensor module 620. In variousembodiments, network module 616 may retrieve specific values, such assensing data 662, connected vehicle data 664, and/or historical data(e.g., previous ACLs, calculations that were computed by remote devices,etc.). For example, user readiness computation module 330 could comparethe current ACL with previous performances before mapping the ACL to adriver readiness value. In some embodiments, the driver readiness 332may include a notification indicating whether the driver has improved,remained more focused, engaged, etc., compared to past performances.

In some embodiments, network module 616 may transmit data acquired byhead unit, such as one or more ACLs, user readiness state 332, and/orsensing data 662 acquired by sensor module 620. In such instances, oneor more devices connected to network 150 may merge data received fromnetwork module 616 with data from other vehicles, and/or infrastructurebefore being consumed by computation modules. For example, one or moredevices may accumulate and compile sensing data in order to associatedriving conditions with required driver readiness.

For example, one or more devices could accumulate multiple ACLcomputations into an aggregate measure of the focus or engagement of agroup. For example, a smart home system that includesaffective-cognitive load based assistant 300 may compute ACLs for eachmember in a room and may generate an aggregate user readiness statecorresponding to the multiple members in the rooms before determiningthe brightness of lighting of a given room. In another example,affective-cognitive load-based assistant 300 could emphasize certainmembers (e.g., emphasize the ACLs of a group of guests).

Output module 340 performs one or more actions in response to a userreadiness state 332 provided by ADAS 618. For example, output module 340may generate one or more output signals 341 in response to userreadiness state 332 that modifies an application and/or interface. Forexample, output module 340 could generate one or more output signals 341to modify HMI 646 to display notification messages and/or alerts. Inanother example, output module 340 could generate one or more outputsignals to modify an application. In such instances, the output signal341 may include application parameters 652 and/or application events654.

In various embodiments, ADAS notifications 642 may include lightindications, such as ambient lights and mood lights, audionotifications, voice notifications (e.g., a voice assistant), visualnotification messages, haptic notifications in the vehicle (e.g.,steering wheel, seat, head rest, etc.) or wearable device or touchlesshaptic notifications, etc.

In various embodiments, ADAS parameters 644 may include variousoperating parameters, settings, or actions. For example, ADAS parameters644 could include vehicle climate control settings (e.g., windowcontrols, passenger compartment temperature, increasing fan speed,etc.), and/or olfactory parameters, such as emitting specific fragrancesthat are calming or stimulating. In various embodiments, ADAS parameters644 may include emergency calling parameters, such as triggering thedialing of one or more emergency phone numbers or suggesting that theuser connect to a specific contact situations that may require immediateassistance and/or response.

In various embodiments, ADAS parameters 644 may dynamically activateL2+/L3+ capabilities, such as lane assist, collision avoidance, and/orautonomous driving. In some embodiments, ADAS parameter 644 may be abinary activation signal (on/off); alternatively, ADAS parameters 644may be activation signal that provides a more-gradual activation (e.g.,with varying degrees of automated correction when the driver seems todeviate from their lane). In some embodiments, ADAS parameters 644 maydynamically activate the collision avoidance systems. For example,output module 340 may dynamically generate ADAS parameters 644 thatadapt the parameters of the system (e.g., warning time, brake intensity,etc.) depending on user readiness state 332.

In some embodiments, other systems may generate responses based on userreadiness state 332. For example, navigation subsystem 614 couldgenerate specific route suggestions based on the user readiness state,such as avoiding routes that require significant focus or attention.Additionally or alternatively, entertainment subsystem 612 may playspecific tracks associated with specific moods in order to maintain userreadiness state 332, or to alter user readiness state 332.

FIG. 7 is a flow diagram of method steps for generating an output signalbased on an affective-cognitive load, according to various embodiments.Although the method steps are described with respect to the systems ofFIGS. 1-6, persons skilled in the art will understand that any systemconfigured to perform the method steps, in any order, falls within thescope of the various embodiments.

As shown, method 700 begins at step 701, where user readinessapplication 140 acquires sensor data. In various embodiments, varioussensor(s) 120 acquire sensor data related to the brain activity and/oremotional state of a user. For example, sensor 120 may include a camerathat acquires sensor data focused on the face of the user.

At step 703, user readiness application 140 computes a cognitive loadfrom the sensor data. In various embodiments, user readiness application140 may analyze portions of the acquired sensor data in order toestimate the cognitive load currently being experienced by a user. Forexample, user readiness application 140 may perform various pupillometrytechniques on received image data in order to determine fluctuations inthe pupil of the user. User readiness application 140 may then computethe cognitive load from the pupil data. Additionally, or alternatively,user readiness application 140 may perform various eye motion analyseson received image data in order to determine eye saccades, eyefixations, and the like, from the eyes of the user. User readinessapplication 140 may then compute the cognitive load from the eyesaccades, eye fixations, and the like.

At step 705, user readiness application 140 computes one or more emotionmetrics from the sensor data. In various embodiments, it may analyzeportions of the acquired sensor data in order to estimate emotionparameterized metrics currently being experienced by a user. Forexample, user readiness application 140 may perform various facialexpression estimation techniques on received image data in order todetermine the emotion being experienced by the user. In another example,additionally or alternatively to facial expression estimation, userreadiness application 140 may perform various voice tone analyses onreceived audio data in order to determine the emotion being experiencedby the user. User readiness application 140 may map the estimatedemotion to specific arousal and valence values.

At step 707, user readiness application 140 determines anaffective-cognitive load (ACL) as a composite of the cognitive load andemotion metrics. In various embodiments, user readiness application 140may execute one or more algorithmic techniques in order to compute anaffective-cognitive load (ACL) of the user. For example, user readinessapplication 140 may compute the ACL as a function of the inverses of thecognitive load and the arousal, multiplied by the valence (e.g.,equation 4). Such an algorithm emphasizes positive emotional valencevalues for a given set of cognitive load and arousal values. In anotherexample, user readiness application 140 could refer to one or morelookup tables to find an entry that specifies a specific ACL for aspecific combination of cognitive load, arousal, and valence.

At step 709, user readiness application 140 compares the ACL to one ormore threshold ACL values in order to determine a user readiness state.In various embodiments, user readiness application 140 may compare theACL value to one or more thresholds that separate user readiness states.For example, user readiness application 140 may compare ACL to a minimumthreshold and a maximum threshold in order to determine that ACL iswithin a target ACL threshold range associated with a particular userreadiness state. In such instances, user readiness application 140 maydetermine that heuristics indicate that the target ACL range correspondsto a medium level of user readiness, indicating that the user is engagedenough to respond to stimuli without being overwhelmed. In otherembodiments, the minimum threshold separate a medium ACL range from alower ACL range that corresponds to the target ACL range. In suchinstances, a medium ACL value could correspond to a user readiness stateoutside the target range, indicating that some assistance to the usermay be necessary to perform the required tasks.

At step 711, user readiness application 140 may optionally cause outputdevice 340 to generate an output signal based on the user readinessstate. In various embodiments, user readiness application 140 may sendto output module 340 the user readiness state. In such instances,receiving the user readiness state causes output module 340 to generateone or more output signals. In various embodiments, the output signalmay cause one or more devices, such as an assisted driving system,display device, and/or feedback system, to adjust one or more parametersassociated with handling a task. For example, upon receiving a userreadiness state indicating that intervention is required, output device340 could generate one or more output signals to assist the user inhandling a task and/or lowering the ACL such that the user will soon beable to successfully handle the task.

In sum, an affective-cognitive based load digital assistant receivessensor data and determines the readiness of a user to handle a taskwithin an environment. Various sensors acquire sensor data associatedwith the user or the environment and sends the sensor data to a userreadiness application included in the affective-cognitive load baseddigital assistant. The user readiness application computes from thesensor data various biometric values associated with thepsychophysiological state of a user. Such biometric values include acognitive load that estimates the amount of brain activity that a useris employing, and an affective load that estimates an emotional loadthat a user is experiencing. In various embodiments, the affective loadmay include one or more separate emotional metrics, such as separatearousal and/or valence values. Upon determining separate biometricvalues for a user, the user readiness application analyzes the cognitiveload and affective load using one or more algorithmic techniques todetermine an affective-cognitive load (ACL), which indicates the user'soverall mental activity. The user readiness application then determinesa user readiness state, which indicates the ability of the user tomanage a set of tasks. The user readiness state determined by the userreadiness application causes output devices and/or other applications toassist the user at a degree that corresponds to the determined userreadiness state.

At least one technological advantage of the affective-cognitive loadbased digital assistant is that computing a user readiness state as acomposite of direct measurements that estimate the cognitive load andthe emotional state of the user provides a more accurate indication ofthe ability of the user to handle tasks. In particular, by combiningemotion recognition and cognitive load, the affective-cognitive loadbased digital assistant can describe both the quality of the cognition(the operations being done by the mind, measured as cognitive load), aswell as the emotion. Further, the mental state estimated by theaffective-cognitive load digital assistant is associated with autonomicbody systems making the estimations less susceptible to error. As aresult, systems that assist the user in conducting certain tasks canmore accurately assist the user based on the estimated brain activityand emotional state of the user.

1. In various embodiments, a computer-implemented method comprisesreceiving, from at least one sensor, sensor data associated with anenvironment, computing, based on the sensor data, a cognitive loadassociated with a user within the environment, computing, based on thesensor data, an affective load associated with an emotional state of theuser, determining, based on both the cognitive load at the affectiveload, an affective-cognitive load, determining, based on theaffective-cognitive load, a user readiness state associated with theuser, and causing one or more actions to occur based on the userreadiness state.

2. The computer-implemented method of clause 1, where the affective loadcomprises an arousal value and a valence value, and determining theaffective-cognitive load comprises searching, based on the cognitiveload, the arousal value, and the valence value, one or more lookuptables that include entries specifying affective-cognitive loads, andidentifying an entry included in the one or more lookup tables, whereinthe entry corresponds to a set of the cognitive load, the arousal value,and the valence value.

3. The computer-implemented method of clause 1 or 2, where the sensordata comprises image data.

4. The computer-implemented method of any of clauses 1-3, where theaffective load comprises an arousal value and a valence value, anddetermining the affective-cognitive load comprises applying analgorithmic technique to compute the affective-cognitive load as afunction of the cognitive load and at least one of the arousal value orthe valence value.

5. The computer-implemented method of any of clauses 1-4, where thealgorithmic technique includes applying a Weibull distribution functionbased on the cognitive load and the arousal value.

6. The computer-implemented method of any of clauses 1-5, wherecomputing the cognitive load comprises determining one or more eyeparameters from image data included in the sensor data, and computingthe cognitive load from the one or more eye parameters, and computingthe affective load comprises determining a pre-defined emotion from thesensor data, and identifying an arousal value corresponding to thepre-defined emotion, and identifying a valence value corresponding tothe pre-defined emotion, where the arousal value and the valence valueare included in the affective load.

7. The computer-implemented method of any of clauses 1-6, where causingthe one or more actions to occur based on the user readiness statecomprises causing an output device to generate an output signal based onthe user readiness state.

8. The computer-implemented method of any of clauses 1-7, where theoutput signal causes the output device to change at least one operatingparameter.

9. The computer-implemented method of any of clauses 1-8, where theoutput signal causes at least one notification message to be emittedfrom the output device.

10. The computer-implemented method of any of clauses 1-9, where theoutput signal causes one or more lights to change brightness.

11. In various embodiments, one or more non-transitory computer-readablemedia store instructions that, when executed by one or more processors,cause the one or more processors to perform the steps of receiving, fromat least one sensor, sensor data associated with an environment,computing, based on the sensor data, a cognitive load associated with auser within the environment, computing, based on the sensor data, anaffective load associated with an emotional state of the user, applyingan algorithmic technique to compute an affective-cognitive load as afunction of the cognitive load and the affective load, determining,based on the affective-cognitive load, a user readiness state associatedwith the user, and causing one or more actions to occur based on theuser readiness state.

12. The one or more non-transitory computer-readable media of clause 11,wherein the sensor data comprises image data.

13. The one or more non-transitory computer-readable media of clause 11or 12, where computing the cognitive load comprises determining one ormore eye parameters from image data included in the sensor data, andcomputing the cognitive load from the one or more eye parameters, andcomputing the affective load comprises determining a pre-defined emotionfrom the sensor data, and identifying an arousal value corresponding tothe pre-defined emotion, and identifying a valence value correspondingto the pre-defined emotion, where the arousal value and the valencevalue are included in the affective load.

14. The one or more non-transitory computer-readable media of any ofclauses 11-13, where the sensor data comprises biometric data includingat least one of a pupil size, a heart rate, a galvanic skin response, ora blood oxygenation level.

15. The one or more non-transitory computer-readable media of any ofclauses 11-14, further including instructions that, when executed by theone or more processors, cause the one or more processors to perform thesteps of computing, based on the sensor data, a second cognitive loadassociated with a second user within the environment, computing, basedon the sensor data, a second affective load associated with a secondemotional state of the second user, determining, based on both thesecond cognitive load at the affective load, a secondaffective-cognitive load, combining the affective-cognitive load and thesecond affective-cognitive load to generate an aggregateaffective-cognitive load, determining, based on the aggregateaffective-cognitive load, an aggregate user readiness state associatedwith both the user and the second user, and causing one or more actionsto occur based on the aggregate user readiness state.

16. In various embodiments, an affective-cognitive load based devicecomprises at least one sensor configured to acquire sensor data, amemory storing a user readiness application, and a processor that iscoupled at least to the memory and, when executing the user readinessapplication, is configured to receive, from the at least one sensor,sensor data associated with an environment, compute, based on the sensordata, a cognitive load associated with a user within the environment,compute, based on the sensor data, at least one of an arousal value or avalence value associated with an emotional state of the user, determine,based on the cognitive load and at least one of the arousal value or thevalence value, an affective-cognitive load, determine, based on theaffective-cognitive load, a user readiness state associated with theuser, and cause one or more actions to occur based on the user readinessstate.

17. The affective-cognitive load-based device of clause 16, where thesensor data comprises image data.

18. The affective-cognitive load-based device of clause 16 or 17, wheredetermining the affective-cognitive load comprises applying analgorithmic technique to compute the affective-cognitive load as afunction of the cognitive load and at least one of the arousal value orthe valence value.

19. The affective-cognitive load-based device of any of clauses 16-18,where the affective-cognitive load-based device is included in anadvanced driver assistance system (ADAS) of a vehicle.

20. The affective-cognitive load-based device of any of clauses 16-19,wherein the processor configured to cause one or more actions to occurcomprises generating an output signal that causes the ADAS to change atleast one ADAS parameter.

Any and all combinations of any of the claim elements recited in any ofthe claims and/or any elements described in this application, in anyfashion, fall within the contemplated scope of the present invention andprotection.

The descriptions of the various embodiments have been presented forpurposes of illustration, but are not intended to be exhaustive orlimited to the embodiments disclosed. Many modifications and variationswill be apparent to those of ordinary skill in the art without departingfrom the scope and spirit of the described embodiments.

Aspects of the present embodiments may be embodied as a system, methodor computer program product. Accordingly, aspects of the presentdisclosure may take the form of an entirely hardware embodiment, anentirely software embodiment (including firmware, resident software,micro-code, etc.) or an embodiment combining software and hardwareaspects that may all generally be referred to herein as a “module,” a“system,” or a “computer.” In addition, any hardware and/or softwaretechnique, process, function, component, engine, module, or systemdescribed in the present disclosure may be implemented as a circuit orset of circuits. Furthermore, aspects of the present disclosure may takethe form of a computer program product embodied in one or more computerreadable medium(s) having computer readable program code embodiedthereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

Aspects of the present disclosure are described above with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general-purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine. The instructions, when executed via the processor ofthe computer or other programmable data processing apparatus, enable theimplementation of the functions/acts specified in the flowchart and/orblock diagram block or blocks. Such processors may be, withoutlimitation, general purpose processors, special-purpose processors,application-specific processors, or field-programmable gate arrays.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present disclosure. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

While the preceding is directed to embodiments of the presentdisclosure, other and further embodiments of the disclosure may bedevised without departing from the basic scope thereof, and the scopethereof is determined by the claims that follow.

What is claimed is:
 1. A computer-implemented method comprising:receiving, from at least one sensor, sensor data associated with anenvironment; computing, based on the sensor data, a cognitive loadassociated with a user within the environment; computing, based on thesensor data, an affective load associated with an emotional state of theuser; determining, based on both the cognitive load at the affectiveload, an affective-cognitive load; determining, based on theaffective-cognitive load, a user readiness state associated with theuser; and causing one or more actions to occur based on the userreadiness state.
 2. The computer-implemented method of claim 1, wherein:the affective load comprises an arousal value and a valence value; anddetermining the affective-cognitive load comprises: searching, based onthe cognitive load, the arousal value, and the valence value, one ormore lookup tables that include entries specifying affective-cognitiveloads, and identifying an entry included in the one or more lookuptables, wherein the entry corresponds to a set of the cognitive load,the arousal value, and the valence value.
 3. The computer-implementedmethod of claim 1, wherein the sensor data comprises image data.
 4. Thecomputer-implemented method of claim 1, wherein: the affective loadcomprises an arousal value and a valence value; and determining theaffective-cognitive load comprises applying an algorithmic technique tocompute the affective-cognitive load as a function of the cognitive loadand at least one of the arousal value or the valence value.
 5. Thecomputer-implemented method of claim 4, wherein the algorithmictechnique includes applying a Weibull distribution function based on thecognitive load and the arousal value.
 6. The computer-implemented methodof claim 1, wherein: computing the cognitive load comprises: determiningone or more eye parameters from image data included in the sensor data,and computing the cognitive load from the one or more eye parameters;and computing the affective load comprises: determining a pre-definedemotion from the sensor data, and identifying an arousal valuecorresponding to the pre-defined emotion, and identifying a valencevalue corresponding to the pre-defined emotion, wherein the arousalvalue and the valence value are included in the affective load.
 7. Thecomputer-implemented method of claim 1, wherein causing the one or moreactions to occur based on the user readiness state comprises causing anoutput device to generate an output signal based on the user readinessstate.
 8. The computer-implemented method of claim 7, wherein the outputsignal causes the output device to change at least one operatingparameter.
 9. The computer-implemented method of claim 7, wherein theoutput signal causes at least one notification message to be emittedfrom the output device.
 10. The computer-implemented method of claim 7,wherein the output signal causes one or more lights to changebrightness.
 11. One or more non-transitory computer-readable mediastoring instructions that, when executed by one or more processors,cause the one or more processors to perform the steps of: receiving,from at least one sensor, sensor data associated with an environment;computing, based on the sensor data, a cognitive load associated with auser within the environment; computing, based on the sensor data, anaffective load associated with an emotional state of the user; applyingan algorithmic technique to compute an affective-cognitive load as afunction of the cognitive load and the affective load; determining,based on the affective-cognitive load, a user readiness state associatedwith the user; and causing one or more actions to occur based on theuser readiness state.
 12. The one or more non-transitorycomputer-readable media of claim 11, wherein the sensor data comprisesimage data.
 13. The one or more non-transitory computer-readable mediaof claim 11, wherein: computing the cognitive load comprises:determining one or more eye parameters from image data included in thesensor data, and computing the cognitive load from the one or more eyeparameters; and computing the affective load comprises: determining apre-defined emotion from the sensor data, and identifying an arousalvalue corresponding to the pre-defined emotion, and identifying avalence value corresponding to the pre-defined emotion, wherein thearousal value and the valence value are included in the affective load.14. The one or more non-transitory computer-readable media of claim 11,wherein the sensor data comprises biometric data including at least oneof a pupil size, a heart rate, a galvanic skin response, or a bloodoxygenation level.
 15. The one or more non-transitory computer-readablemedia of claim 11, further including instructions that, when executed bythe one or more processors, cause the one or more processors to performthe steps of: computing, based on the sensor data, a second cognitiveload associated with a second user within the environment; computing,based on the sensor data, a second affective load associated with asecond emotional state of the second user; determining, based on boththe second cognitive load at the affective load, a secondaffective-cognitive load; combining the affective-cognitive load and thesecond affective-cognitive load to generate an aggregateaffective-cognitive load; determining, based on the aggregateaffective-cognitive load, an aggregate user readiness state associatedwith both the user and the second user; and causing one or more actionsto occur based on the aggregate user readiness state.
 16. Anaffective-cognitive load-based device, comprising: at least one sensorconfigured to acquire sensor data; a memory storing a user readinessapplication; and a processor that is coupled at least to the memory and,when executing the user readiness application, is configured to:receive, from the at least one sensor, sensor data associated with anenvironment; compute, based on the sensor data, a cognitive loadassociated with a user within the environment; compute, based on thesensor data, at least one of an arousal value or a valence valueassociated with an emotional state of the user; determine, based on thecognitive load and at least one of the arousal value or the valencevalue, an affective-cognitive load; determine, based on theaffective-cognitive load, a user readiness state associated with theuser; and cause one or more actions to occur based on the user readinessstate.
 17. The affective-cognitive load-based device of claim 16,wherein the sensor data comprises image data.
 18. Theaffective-cognitive load-based device of claim 16, wherein determiningthe affective-cognitive load comprises applying an algorithmic techniqueto compute the affective-cognitive load as a function of the cognitiveload and at least one of the arousal value or the valence value.
 19. Theaffective-cognitive load-based device of claim 16, wherein theaffective-cognitive load-based device is included in an advanced driverassistance system (ADAS) of a vehicle.
 20. The affective-cognitiveload-based device of claim 19, wherein the processor configured to causeone or more actions to occur comprises generating an output signal thatcauses the ADAS to change at least one ADAS parameter.